Skip to content

feat(outcome): build verifier-bound skill scorecards - #591

Merged
solomonneas merged 7 commits into
mainfrom
feat/502-verifier-scorecards
Jul 27, 2026
Merged

feat(outcome): build verifier-bound skill scorecards#591
solomonneas merged 7 commits into
mainfrom
feat/502-verifier-scorecards

Conversation

@solomonneas

Copy link
Copy Markdown
Collaborator

Summary

Implements the scorecard design from #570 and completes #502 through its five ordered slices:

  • accepts score inputs only from tracked, verifier-authored manifests with bound subjects, check roles, patch or fixture identity, receipt digests, and failure taxonomy
  • projects current-fingerprint effectiveness, verifier cost, retry stability, evidence integrity, and per-check utility retention from eligible verify receipts
  • gives unseen and low-data skills deterministic, route-scoped exploration with 7-day, 30-day, per-run, and optional budget caps
  • promotes only when effectiveness and every required utility check pass, then records scorecard.v1 routing authority; one trusted hurt removes that authority immediately
  • adds read-only scorecard backfill, work-brief fields, and operator checkup --surface outcome

Why

The previous loop counted distinct ledger artifact ids as scored, relied on optional caller-attributed capture, exposed rank as a read-only view, and left reconcile dry-run by default. Receipt volume could therefore grow without producing promotion-ready evidence. Caller labels and no-work receipts could also contaminate routing if treated as score inputs.

This change keeps legacy records.jsonl rows audit-only for skill promotion. It does not infer subjects from those rows or convert them during backfill.

Compatibility

Cards remain on the existing ledger decision path. Generated-patch evidence must satisfy verifier independence, and required utility checks remain verifier-manifest declarations, so this composes with #507 and #499 without making their audits blocking.

Verification

  • brigade work verify run --target . --command "./scripts/verify" --capture brigade-work
  • rebased receipt: .brigade/work/verify-runs/20260727-002601-work-verify-66ea63/receipt.json
  • 4,598 passed, 3 skipped
  • coverage: 82.91%

Closes #502
Closes #571
Closes #572
Closes #573
Closes #503
Closes #574

solomonneas and others added 7 commits July 26, 2026 20:25
Co-Authored-By: Cursor <cursoragent@cursor.com>
Co-Authored-By: Cursor <cursoragent@cursor.com>
Co-Authored-By: Cursor <cursoragent@cursor.com>
Co-Authored-By: Cursor <cursoragent@cursor.com>
Co-Authored-By: Cursor <cursoragent@cursor.com>
Co-Authored-By: Cursor <cursoragent@cursor.com>
Co-Authored-By: Cursor <cursoragent@cursor.com>
@coderabbitai

coderabbitai Bot commented Jul 27, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are limited based on label configuration.

🏷️ Required labels (at least one) (1)
  • coderabbit-review

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository: escoffier-labs/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 8b2bc37c-ffa8-4c5f-8f79-1e37b28acfcf

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/502-verifier-scorecards

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@solomonneas solomonneas self-assigned this Jul 27, 2026
@solomonneas
solomonneas merged commit c783d89 into main Jul 27, 2026
28 checks passed
@solomonneas
solomonneas deleted the feat/502-verifier-scorecards branch July 27, 2026 00:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment